fix!: build judge input as string and strip legacy judge config messages#1364
Merged
Merged
Conversation
Contributor
|
@launchdarkly/js-sdk-common size report |
Contributor
|
@launchdarkly/js-client-sdk size report |
Contributor
|
@launchdarkly/browser size report |
Contributor
|
@launchdarkly/js-client-sdk-common size report |
jsonbailey
force-pushed
the
jb/fix-judge-string-input
branch
3 times, most recently
from
e4e5069 to
f4960cc
Compare
jsonbailey
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f4960cc. Configure here.
3 tasks
Aligns the JS AI SDK with the spec change implemented in launchdarkly/python-server-sdk-ai#165 and launchdarkly/sdk-specs#160. Judges now build a single formatted string ("MESSAGE HISTORY:\n...\n\n RESPONSE TO EVALUATE:\n...") and pass it to the runner instead of an interpolated message list. Legacy judge configs that contain {{message_history}} or {{response_to_evaluate}} placeholders in non-system messages are stripped at config-construction time so old and new flag values both work without behavioral surprises. BREAKING CHANGE: Runner.run is narrowed from `run(input: string | LDMessage[], outputType?)` to `run(input: string, outputType?)`. The OpenAI, LangChain, and Vercel provider runners no longer accept a pre-built message array; they always prepend any config messages and append the prompt as a user turn. The Judge no longer interpolates {{message_history}} or {{response_to_evaluate}} into config messages — the SDK builds the input string directly and the runner receives that string verbatim. The "Judge configuration must include messages" early-return was removed; a judge with no messages now proceeds to invoke the runner with the formatted input. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jsonbailey
force-pushed
the
jb/fix-judge-string-input
branch
from
f4960cc to
cfb0438
Compare
joker23
approved these changes
May 7, 2026
jsonbailey
added a commit
that referenced
this pull request
May 7, 2026
Merged
jsonbailey
pushed a commit
that referenced
this pull request
May 7, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>server-sdk-ai: 0.20.0</summary> ## [0.20.0](server-sdk-ai-v0.19.1...server-sdk-ai-v0.20.0) (2026-05-07) ### ⚠ BREAKING CHANGES * Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens ([#1366](#1366)) * Remove AIProvider deprecated methods and create*/init* aliases (AIC-2388) ([#1363](#1363)) * Build judge input as string and strip legacy judge config messages ([#1364](#1364)) * Use LDAIGraphMetricSummary for graph metric summary ([#1362](#1362)) ### Features * add Evaluator class for judge orchestration ([#1331](#1331)) ([54faa69](54faa69)) * add ManagedAgent with evaluations support ([#1334](#1334)) ([7f09c46](7f09c46)) * add ManagedGraphResult, GraphMetricSummary, and ManagedAgentGraph ([#1335](#1335)) ([09fa1db](09fa1db)) * introduce ManagedResult, RunnerResult, and LDAIMetricSummary ([#1332](#1332)) ([5040122](5040122)) * Remove AIProvider deprecated methods and create*/init* aliases (AIC-2388) ([#1363](#1363)) ([ad66314](ad66314)) * Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens ([#1366](#1366)) ([ff932b7](ff932b7)) * Replace OpenAIProvider with Runner protocol implementation (AIC-2388) ([#1337](#1337)) ([e32a955](e32a955)) ### Bug Fixes * Build judge input as string and strip legacy judge config messages ([#1364](#1364)) ([c90034b](c90034b)) * Use LDAIGraphMetricSummary for graph metric summary ([#1362](#1362)) ([76a4bf2](76a4bf2)) </details> <details><summary>server-sdk-ai-langchain: 0.7.0</summary> ## [0.7.0](server-sdk-ai-langchain-v0.6.3...server-sdk-ai-langchain-v0.7.0) (2026-05-07) ### ⚠ BREAKING CHANGES * Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens ([#1366](#1366)) * Remove AIProvider deprecated methods and create*/init* aliases (AIC-2388) ([#1363](#1363)) * Build judge input as string and strip legacy judge config messages ([#1364](#1364)) ### Features * Remove AIProvider deprecated methods and create*/init* aliases (AIC-2388) ([#1363](#1363)) ([ad66314](ad66314)) * Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens ([#1366](#1366)) ([ff932b7](ff932b7)) * Replace LangChainProvider with Runner protocol implementation (AIC-2388) ([#1338](#1338)) ([113a0d2](113a0d2)) ### Bug Fixes * Build judge input as string and strip legacy judge config messages ([#1364](#1364)) ([c90034b](c90034b)) ### Dependencies * The following workspace dependencies were updated * devDependencies * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0 * peerDependencies * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0 </details> <details><summary>server-sdk-ai-openai: 0.6.0</summary> ## [0.6.0](server-sdk-ai-openai-v0.5.10...server-sdk-ai-openai-v0.6.0) (2026-05-07) ### ⚠ BREAKING CHANGES * Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens ([#1366](#1366)) * Remove AIProvider deprecated methods and create*/init* aliases (AIC-2388) ([#1363](#1363)) * Build judge input as string and strip legacy judge config messages ([#1364](#1364)) ### Features * Remove AIProvider deprecated methods and create*/init* aliases (AIC-2388) ([#1363](#1363)) ([ad66314](ad66314)) * Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens ([#1366](#1366)) ([ff932b7](ff932b7)) * Replace OpenAIProvider with Runner protocol implementation (AIC-2388) ([#1337](#1337)) ([e32a955](e32a955)) ### Bug Fixes * Build judge input as string and strip legacy judge config messages ([#1364](#1364)) ([c90034b](c90034b)) ### Dependencies * The following workspace dependencies were updated * devDependencies * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0 * peerDependencies * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0 </details> <details><summary>server-sdk-ai-vercel: 0.6.0</summary> ## [0.6.0](server-sdk-ai-vercel-v0.5.10...server-sdk-ai-vercel-v0.6.0) (2026-05-07) ### ⚠ BREAKING CHANGES * Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens ([#1366](#1366)) * Remove AIProvider deprecated methods and create*/init* aliases (AIC-2388) ([#1363](#1363)) * Build judge input as string and strip legacy judge config messages ([#1364](#1364)) ### Features * Remove AIProvider deprecated methods and create*/init* aliases (AIC-2388) ([#1363](#1363)) ([ad66314](ad66314)) * Rename LDAIMetrics.usage and LDAIGraphMetrics.usage to .tokens ([#1366](#1366)) ([ff932b7](ff932b7)) * replace VercelProvider with Runner protocol implementation (AIC-2388) ([#1339](#1339)) ([d5a62de](d5a62de)) ### Bug Fixes * add zod devDependency to Vercel provider (peer dep of ai v5) ([aab6226](aab6226)) * Build judge input as string and strip legacy judge config messages ([#1364](#1364)) ([c90034b](c90034b)) ### Dependencies * The following workspace dependencies were updated * devDependencies * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0 * peerDependencies * @launchdarkly/server-sdk-ai bumped from ^0.19.1 to ^0.20.0 </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > This is primarily a versioning/release metadata update, but it ships breaking API changes in `@launchdarkly/server-sdk-ai` and provider packages that may require consumer code updates (e.g., metrics field rename and deprecated method removals). > > **Overview** > Publishes a new release for `@launchdarkly/server-sdk-ai` (`0.19.1` → `0.20.0`) and the provider packages (`server-sdk-ai-langchain` `0.6.3` → `0.7.0`, `server-sdk-ai-openai`/`server-sdk-ai-vercel` `0.5.10` → `0.6.0`), updating `.release-please-manifest.json`, package versions, and `sdkInfo.ts`. > > Updates provider `peerDependencies`/`devDependencies` to require `@launchdarkly/server-sdk-ai@^0.20.0`, and bumps all `server-ai` examples to depend on the new SDK/provider versions. Changelogs highlight *breaking changes* including renaming metrics `usage` → `tokens`, removing deprecated `AIProvider` methods/aliases, and judge-input/config cleanup (plus additional `server-sdk-ai` graph/managed result summary changes). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit cb3aebc. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MESSAGE HISTORY:\n...\n\nRESPONSE TO EVALUATE:\n...) to the runner instead of an interpolated message list.{{message_history}}/{{response_to_evaluate}}placeholder messages are stripped at config construction so old and new flag values continue to work.Runner.runis narrowed to(input: string, outputType?)everywhere — server-ai, server-ai-openai, server-ai-langchain, server-ai-vercel.Breaking changes
Runner.run(input: string | LDMessage[], outputType?)→Runner.run(input: string, outputType?). Provider runners (OpenAI, LangChain, Vercel) no longer accept a pre-builtLDMessage[]. They always prepend any config messages and append the prompt as a user turn.Judgeno longer Mustache-interpolates{{message_history}}/{{response_to_evaluate}}into config messages. The SDK builds the input string and passes it verbatim to the runner.Judge.evaluateno longer early-returns with"Judge configuration must include messages"— a judge with no messages now proceeds to invoke the runner with the formatted input string.Test plan
yarn workspace @launchdarkly/server-sdk-ai test— 239 passingyarn workspace @launchdarkly/server-sdk-ai-openai test— 39 passingyarn workspace @launchdarkly/server-sdk-ai-langchain test— 38 passingyarn workspace @launchdarkly/server-sdk-ai-vercel test— 20 passinglintclean for all four packagesyarn workspaces foreach -pR --topological-dev --from '@launchdarkly/server-sdk-ai' run build— all passchat-judgeexample end-to-end and confirm judge events emit correctly🤖 Generated with Claude Code
Note
Medium Risk
Medium risk due to a breaking API change (
Runner.runno longer acceptsLDMessage[]) and behavioral changes in judge evaluation/config processing that could affect downstream integrations.Overview
Breaking API change:
Runner.runis narrowed from(string | LDMessage[])to(string)acrossserver-aiand provider packages, and provider runners (OpenAI/LangChain/Vercel) now always build the message list by prepending config messages and appending the prompt as a user turn.Judge evaluation update:
Judgestops Mustache-interpolating legacy templates into message arrays and instead builds a single evaluation string (MESSAGE HISTORY... / RESPONSE TO EVALUATE...) that it passes to the runner; it also no longer errors whenjudgeConfig.messagesis missing.Backward compatibility:
LDAIClientImpl.judgeConfignow strips legacy judge placeholder messages containing{{message_history}}/{{response_to_evaluate}}before returning configs, with tests updated/added to cover the new input format and stripping behavior.Reviewed by Cursor Bugbot for commit f4960cc. Bugbot is set up for automated code reviews on this repo. Configure here.